Image processing apparatus and image processing method

ABSTRACT

A deblocking filter applies filtering to reduce blocking artifacts from an encoded stream in which image data has been encoded in individual blocks. A filter strength adjuster adjusts the filter strength according to block sizes of the predictive image data. For example, during image encoding, the filter strength is adjusted according to the block size of the predictive image data yielding the best encoding efficiency generated using filtered and decoded image data. Filtering may also be applied to remove blocking artifacts from decoded image data during image decoding, with the filter strength being adjusted according to the block sizes of the predictive image data. Decoded images with favorable image quality are obtained, even in the case of using extended macroblock sizes.

TECHNICAL FIELD

This invention relates to an image processing apparatus and an imageprocessing method. Specifically, it is configured such that a decodedimage with favorable image quality is obtained.

BACKGROUND ART

Recently, there has been a proliferation in broadcasting stations andgeneral households of apparatus that digitally handle image information,and when so doing, compress images for the purpose of efficientinformation transfer and storage. Such apparatus compress images byimplementing coding formats such as MPEG which compress informationusing an orthogonal transform such as the discrete cosine transform andby motion compensation.

Particularly, MPEG-2 (ISO/IEC 13818-2) is defined as a general-purposeimage coding format, and is currently widely used in a broad range ofapplications for both professional use and consumer use. By using theMPEG-2 compression format, it is possible to realize favorable imagequality by allocating a bit rate from 4 to 8 Mbps if given astandard-definition interlaced image having 720×480 pixels, for example.Additionally, it is possible to realize favorable image quality byallocating a bit rate from 18 to 22 Mbps if given a high-definitioninterlaced image having 1920×1088 pixels.

Although MPEG-2 has primarily targeted high image quality codingsuitable for broadcasting, it is not compatible with coding formatshaving a bit rate lower than that of MPEG-1, or in other words a highcompression rate. Due to the proliferation of mobile devices, it isthought that the need for such coding formats will increase in thefuture, and in response the MPEG-4 coding format has been standardized.MPEG-4 was designated an international standard for image coding inDecember 1998 as ISO/IEC 14496-2.

Furthermore, standardization of H.26L (ITU-T Q6/16 VCEG), which wasinitially for the purpose of image coding for videoconferencing, hasbeen progressing recently. Compared to previous coding formats such asMPEG-2 and MPEG-4, H.26L is known to impose more computational demandsfor coding and decoding, but higher encoding efficiency is realized.Also, as a link to MPEG-4 activity, a format based on H.26L whichrealizes higher encoding efficiency is being currently standardized asthe Joint Model of Enhanced-Compression Video Coding. As part of thestandardization schedule, H.264 and MPEG-4 Part 10 (Advanced VideoCoding, hereinafter abbreviated “H.264/AVC”) was internationallystandardized in March 2003.

Additionally, as an extension of the above, standardization of the FRExt(Fidelity Range Extension) was completed in February 2005. FRExtincludes coding tools required for business use, such as RGB, 4:2:2, and4:4:4, as well as the 8×8 DCT and quantization matrices defined inMPEG-2. As a result, H.264/AVC can be used for image coding able tofavorably express even the film noise included in movies, which has ledto its use in a wide range of applications such as Blu-Ray (trademark).

With such coding and decoding processes, image data is coded in units ofblocks. Also, when decoding coded data, a filter is applied anddeblocking is conducted on the basis of the boundary strength and thequantization parameters, as indicated in PTL 1, for example.

However, needs are growing for coding at even higher compression rates,such as for compressing images having approximately 4000×2000 pixels, orfor delivering high-definition images in an environment of limitedtransmission capacity such as the Internet. Thus, extending themacroblock size beyond that of MPEG-2 and H.264/AVC to a size such as32×32 pixels, for example, has been proposed in literature such asNPL 1. Namely, with the proposal in NPL 1, by adopting a tieredmacroblock structure, larger blocks are defined as a superset whilemaintaining compatibility with the H.264/AVC format for blocks of 16×16pixels or less.

CITATION LIST Patent Literature

-   PTL 1: Japanese Unexamined Patent Application Publication No.    2007-36463

Non Patent Literature

-   NPL 1: “Video Coding Using Extended Block”, (Study Group 16,    Contribution 123, ITU, January 2009)

SUMMARY OF INVENTION Technical Problem

However, blocking artifacts increase in the case of using extendedmacroblock sizes, particularly at low bit rates. For this reason, if aconventional deblocking filter is used to remove deblocking artifacts,there is a risk that the blocking artifacts will not be sufficientlyremoved, resulting in poorer image quality.

Accordingly, it is an object of the present invention to provide animage processing apparatus and image processing method by which decodedimages with favorable image quality are obtained, even in the case ofusing extended macroblock sizes.

Solution to Problem

A first aspect of the present invention is an image processing apparatusprovided with a decoder that generates decoded image data by decoding anencoded stream in which image data has been encoded in individualblocks, a filter that applies filtering to remove blocking artifactsfrom the decoded image data generated by the decoder, and a filterstrength adjuster that adjusts the filter strength of the filteringaccording to the block sizes of neighboring blocks adjacent at a blockboundary.

A second aspect of the present invention is an image processingapparatus provided with a filter that applies filtering to removeblocking artifacts from decoded image data, a filter strength adjusterthat adjusts the filter strength of the filtering according to the blocksizes of neighboring blocks adjacent at a block boundary, and an encoderthat encodes image data by using decoded image data that has beenfiltered by the filter.

With this invention, filtering to remove blocking artifacts is appliedto decoded image data that has been obtained by inverse quantizingquantized data generated by the encoding of image data and applying aninverse orthogonal transform to the inverse quantized data, or todecoded image data that has been obtained by inverse quantizingquantized data obtained by losslessly decoding an encoding stream andapplying an inverse orthogonal transform to the inverse quantized data,for example. With the filtering, the filter strength is adjusted suchthat filtering is more readily applied to larger block sizes. With thefilter strength adjustment, parameter values for adjusting the filterstrength are configured according to block sizes. Also, the filterstrength is adjusted according to the larger block size in the casewhere two neighboring blocks have different block sizes. Also, with thefilter strength adjustment, the values of syntax elements in a codingstandard are adjusted. For example, the filter strength may be adjustedby adjusting the values of FilterOffsetA and FilterOffsetB, which arespecified by the syntax elements slice_alpha_c0_offset_div2 andslice_beta_offset_div2 in the H.264/AVC standard. Furthermore, whenusing parameter values that differ from preset parameter values, thevalue of the syntax element deblocking_filter_control_present_flag inthe picture parameter set is set to a given value such as “1”, andvalues for FilterOffsetA and FilterOffsetB are stated in the sliceheader.

A third aspect of the present invention is an image processing methodprovided with a step that generates decoded image data by decoding anencoded stream in which image data has been encoded in individualblocks, a step that applies filtering to remove blocking artifacts fromthe decoded image data generated by the decoder, and a step that adjuststhe filter strength of the filtering according to the block sizes ofneighboring blocks adjacent at a block boundary.

A fourth aspect of the present invention is an image processing methodprovided with a step that applies filtering to remove blocking artifactsfrom decoded image data, a step that adjusts the filter strength of thefiltering according to the block sizes of neighboring blocks adjacent ata block boundary, and a step that encodes image data by using decodedimage data that has been filtered by the filter.

Advantageous Effects of Invention

According to the invention, the filter strength is adjusted according tothe block sizes of neighboring blocks adjacent at a block boundary whenapplying filtering to remove blocking artifacts from decoded image data.For this reason, the filter strength is adjusted according to the blocksize even if macroblocks of extended size are used, and thus a decodedimage with reduced blocking artifacts and favorable image quality isobtained.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a diagram illustrating a configuration of an image encodingapparatus.

FIG. 2 is a diagram illustrating pixel data used for filtering by adeblocking filter.

FIG. 3 is a diagram illustrating the relationship between a quantizationparameter QP and a threshold value α.

FIG. 4 is a diagram illustrating a configuration of a deblocking filterand a filter strength adjuster.

FIG. 5 is a diagram illustrating prediction block sizes used in an imageencoding process.

FIG. 6 is a diagram illustrating exemplary CUs defined by the HEVCcoding format.

FIG. 7 is a flowchart illustrating image encoding processing operations.

FIG. 8 is a flowchart illustrating a prediction process.

FIG. 9 is a flowchart illustrating an intra prediction process.

FIG. 10 is a flowchart illustrating an inter prediction process.

FIG. 11 is a flowchart illustrating a filter configuration process.

FIG. 12 illustrates a configuration of an image decoding apparatus.

FIG. 13 is a flowchart illustrating image decoding processingoperations.

DESCRIPTION OF EMBODIMENTS

Hereinafter, embodiments for carrying out the present invention will bedescribed. An image processing apparatus of the present invention isapplicable to an image encoding apparatus that encodes image data on aper-block basis, and to an image decoding apparatus that decodes anencoded stream in which image data has been encoded on a per-blockbasis. The case of an image processing apparatus of the presentinvention being applied to an image processing apparatus and the case ofbeing applied to an image decoding apparatus will be described in thefollowing order.

1. Configuration of image encoding apparatus

2. Filtering by deblocking filter

3. Configuration of deblocking filter and filter strength adjuster inimage encoding apparatus

4. Operation of image encoding apparatus

5. Configuration of image decoding apparatus

6. Operation of image decoding apparatus

<1. Configuration of Image Encoding Apparatus>

FIG. 1 illustrates a configuration of an image encoding apparatus. Theimage encoding apparatus 10 is provided with an analog/digital converter(A/D converter) 11, a frame sort buffer 12, a subtractor 13, anorthogonal transform unit 14, a quantizer 15, a lossless encoder 16, anaccumulation buffer 17, and a rate controller 18. Additionally, theimage encoding apparatus 10 is provided with an inverse quantizer 21, aninverse orthogonal transform unit 22, an adder 23, a deblocking filter24, frame memory 25, a selector 26, an intra predictor 31, a motionprediction/compensation unit 32, and a predictive image/optimal modeselector 33.

The A/D converter 11 converts an analog image signal into digital imagedata, which is output to the frame sort buffer 12.

The frame sort buffer 12 sorts image data output from the A/D converter11 in a frame order. The frame sort buffer 12 sorts frames according tothe GOP (Group of Pictures) structure given by the encoding process, andoutputs sorted image data to the subtractor 13, the intra predictor 31,and the motion prediction/compensation unit 32.

The subtractor 13 is supplied with image data output from the frame sortbuffer 12 and predictive image data selected by the predictiveimage/optimal mode selector 33 discussed later. The subtractor 13computes prediction error data, being the difference between the imagedata output from the frame sort buffer 12 and the predictive image datasupplied by the predictive image/optimal mode selector 33, and outputsthe prediction error data to the orthogonal transform unit 14.

The orthogonal transform unit 14 applies an orthogonal transform such asthe discrete cosine transform (DCT) or the Karhunen-Loeve transform tothe prediction error data output from the subtractor 13. The orthogonaltransform unit 14 outputs transform coefficient data obtained by theorthogonal transform to the quantizer 15.

The quantizer 15 is supplied with transform coefficient data output fromthe orthogonal transform unit 14 and a rate control signal from the ratecontroller 18 discussed later. The quantizer 15 quantizes the transformcoefficient data, and outputs quantized data to the lossless encoder 16and the inverse quantizer 21. The quantizer 15 also switches aquantization parameter (quantization scale) on the basis of a ratecontrol signal from the rate controller 18, thereby changing the bitrate of quantized data.

The lossless encoder 16 is supplied with quantized data output from thequantizer 15 and prediction mode information from the intra predictor31, the motion prediction/compensation unit 32, and the predictiveimage/optimal mode selector 33 discussed later. Herein, prediction modeinformation may include a macroblock type by which the prediction blocksize can be identified, a prediction mode, motion vector information,and reference picture information, for example, depending on whether theprediction mode is intra prediction or inter prediction. The losslessencoder 16 losslessly encodes quantized data by variable-length encodingor arithmetic coding, for example, and generates an encoded stream whichis output to the accumulation buffer 17. The lossless encoder 16 alsolosslessly encodes prediction mode information, which is added to theencoded stream header information.

The accumulation buffer 17 buffers the encoded stream from the losslessencoder 16. The accumulation buffer 17 also outputs the buffered encodedstream at a transmission rate that depends on the transmission channel.

The rate controller 18 monitors the amount of free space in theaccumulation buffer 17, generates a rate control signal depending on theamount of free space, and outputs the rate control signal to thequantizer 15. The rate controller 18 acquires information indicating theamount of free space from the accumulation buffer 17, for example. Whenthe amount of free space runs low, the rate controller 18 lowers the bitrate of quantized data with a rate control signal. Also, when there is asufficiently large amount of free space in the accumulation buffer 17,the rate controller 18 raises the bit rate of quantized data with a ratecontrol signal.

The inverse quantizer 21 inverse quantizes quantized data supplied fromthe quantizer 15. The inverse quantizer 21 outputs transform coefficientdata obtained by the inverse quantization to the inverse orthogonaltransform unit 22.

The inverse orthogonal transform unit 22 applies an inverse orthogonaltransform to the transform coefficient data supplied from the inversequantizer 21, and outputs the data obtained thereby to the adder 23.

The adder 23 adds together data from the inverse orthogonal transformunit 22 and predictive image data supplied from the predictiveimage/optimal mode selector 33 to generate decoded image data, which isoutput to the deblocking filter 24 and the frame memory 25.

The deblocking filter 24 applies a filter to reduce blocking artifactsproduced during image encoding. The deblocking filter 24 applies afilter to remove blocking artifacts from the decoded image data suppliedfrom the adder 23, and outputs the filtered and decoded image data tothe frame memory 25. The deblocking filter 24 also adjusts the filterstrength on the basis of parameter values supplied from a filterstrength adjuster 41 discussed later.

The frame memory 25 stores the decoded image data supplied from theadder 23 and the filtered and decoded image data supplied from thedeblocking filter 24.

The selector 26 supplies unfiltered decoded image data read out from theframe memory 25 to the intra predictor 31 for intra prediction. Theselector 26 also supplies filtered decoded image data read out from theframe memory 25 to the motion prediction/compensation unit 32 for interprediction.

The intra predictor 31 conducts an intra prediction process in allcandidate intra prediction modes using image data of the encoding targetimage output from the frame sort buffer 12 as well as unfiltered decodedimage data read out from the frame memory 25. In addition, the intrapredictor 31 computes a cost function value for each intra predictionmode, and selects the intra prediction mode yielding the minimumcomputed cost function value, or in other words the intra predictionmode yielding the best encoding efficiency, as the optimal intraprediction mode. The intra predictor 31 outputs predictive image datagenerated with the optimal intra prediction mode, prediction modeinformation regarding the optimal intra prediction mode, and the costfunction value of the optimal intra prediction mode to the predictiveimage/optimal mode selector 33. Also, in order to obtain the bit rateused in the computation of the cost function value as discussed later,the intra predictor 31 outputs prediction mode information regarding theintra prediction mode to the lossless encoder 16 when intra predictingin each intra prediction mode.

The motion prediction/compensation unit 32 applies motionprediction/compensation to all prediction block sizes corresponding tomacroblocks. The motion prediction/compensation unit 32 uses thefiltered decoded image read out from the frame memory 25 to detect amotion vector for each image block given by the respective predictionblock sizes in an encoding target image read out from the frame sortbuffer 12. Additionally, the motion prediction/compensation unit 32applies motion compensation to the decoded image on the basis of thedetected motion vectors to generate a predictive image. The motionprediction/compensation unit 32 also computes a cost function value foreach prediction block size, and selects the prediction block sizeyielding the minimum computed cost function value, or in other words theprediction block size yielding the best encoding efficiency, as theoptimal inter prediction mode. The motion prediction/compensation unit32 outputs predictive image data generated with the optimal interprediction mode, prediction mode information regarding the optimal interprediction mode, and the cost function value of the optimal interprediction mode to the predictive image/optimal mode selector 33. Also,in order to obtain the bit rate used in the computation of the costfunction value as discussed later, the motion prediction/compensationunit 32 outputs prediction mode information regarding the interprediction mode to the lossless encoder 16 when inter predicting witheach prediction block size. Note that the motion prediction/compensationunit 32 also predicts using skipped macroblocks and direct mode as interprediction modes.

The predictive image/optimal mode selector 33 compares cost functionvalues supplied from the intra predictor 31 to cost function valuessupplied from the motion prediction/compensation unit 32 at themacroblock level, and selects the prediction mode with the lower costfunction value as the optimal mode with the best encoding efficiency.The predictive image/optimal mode selector 33 also outputs predictiveimage data generated with the optimal mode to the subtractor 13 and theadder 23. Additionally, the predictive image/optimal mode selector 33outputs prediction mode information regarding the optimal mode to thelossless encoder 16 and the filter strength adjuster 41. Herein, thepredictive image/optimal mode selector 33 conducts intra prediction orinter prediction at the slice level.

The filter strength adjuster 41 configures parameter values foradjusting the filter strength according to the prediction block sizeindicated by the prediction mode information regarding the optimal mode,and outputs the configured parameter values to the deblocking filter 24.

<2. Filtering by Deblocking Filter>

When filtering with a deblocking filter it is possible to specify threetypes of filtering:

(a) filter block boundaries as well as macroblock boundaries

(b) filter macroblock boundaries only

(c) do not filter

These three types of filtering are specified according to two parametersin the H.264/AVC coding format, deblocking_filter_control_present_flagin the Picture Parameter Set RBSP and disable_deblocking_filter_idc inthe slice header included in the image compression information.

Regarding the quantization parameter QP, QPY is used in the case ofapplying the following process to luminance data, whereas QPC is used inthe case of applying to chrominance data. Also, for motion vectorcoding, intra prediction, and entropy coding (CAVLC/CABAC), pixel valuesbelonging to different slices are processed as “not available”.Furthermore, for filtering, pixel values are processed as “available” ifthey belong to the same picture, even if the pixel values belong todifferent slices.

In the following explanation, unfiltered pixel data at the blockboundary between neighboring blocks P and Q are taken to be p0 to p3 andq0 to q3, as illustrated by (A) in FIG. 2. Also, filtered pixel data istaken to be p0′ to p3′ and q0′ to q3′, as illustrated by (B) in FIG. 2.

Prior to filtering, block boundary strength data Bs (Boundary Strength)is defined as indicated in Table 1 for the pixels p and the pixels q inFIG. 2.

TABLE 1 Either p or q belongs to intra macroblock, Bs = 4 and inaddition, is positioned at the (Strongest macroblock boundary.Filtering) Either p or q belongs to intra macroblock, Bs = 3 but is notpositioned at the macroblock boundary. Neither p nor q belongs to intramacroblock, Bs = 2 and in addition, either p or q has a transformcoefficient. Neither p nor q belongs to intra macroblock Bs = 1 andneither has a transform coefficient; however, the references framesdiffer, the number of reference frames differs, or the MV values differ.Neither q nor q belongs to intra macroblock Bs = 0 and neither has atransform coefficient. (No Reference frames and MV values are identical.Filtering)

As illustrated in Table 1, the block boundary strength data Bs isassigned the highest filter strength of “4” in the case where either apixel p or a pixel q belongs to the macroblock MB to be intra coded, andin addition, that pixel is positioned at the boundary of the macroblockMB.

The block boundary strength data Bs is assigned a filter strength of“3”, the next-highest after “4”, in the case where either a pixel p or apixel q belongs to the macroblock MB to be intra coded, but in addition,that pixel is not positioned at the boundary of the macroblock MB.

The block boundary strength data Bs is assigned a filter strength of“2”, the next-highest after “3”, in the case where neither a pixel p nora pixel q belongs to the macroblock MB to be intra coded, and inaddition, some of the pixels have transform coefficients.

The block boundary strength data Bs is assigned a filter strength of “1”in the case of satisfying a condition wherein neither a pixel p nor apixel q belongs to the macroblock MB to be intra coded, and in addition,none of the pixels have transform coefficients, while also satisfying acondition wherein the reference frames differ, the number of referenceframes differ, or the motion vectors differ.

The block boundary strength data Bs is assigned a filter strength of“0”, meaning that filtering is not applied, in the case where neither apixel p nor a pixel q belongs to the macroblock MB to be intra coded andneither have transform coefficients, but the reference frames and motionvectors are the same.

Filtering is applied to (p2, p1, p0, q0, q1, q2) in FIG. 2 only in thecase where the condition in Eq. (1) is established.

Bs>0

|p0−q0|<α; |p1−p0|β; |q1−q0|<β  (1)

Herein, the threshold values α and β are parameter values that adjustthe filter strength, or in other words, how readily the filter isapplied. By default, the values of α and β are determined according tothe quantization parameter QP as follows. It is also possible for theuser to adjust the strength with the two parametersslice_alpha_c0_offset_div2 and slice_beta_offset_div2 included in theslice header of the image compression information. Herein, FIG. 3illustrates the relationship between the quantization parameter QP andthe threshold value α. FIG. 3 demonstrates how the filter strength isadjusted because of the way the curve, which illustrates therelationship between the quantization parameter QP and the thresholdvalue α, moves in the direction of the arrow as an offset is added tothe quantization parameter QP.

Also, the threshold values α and β are taken from the table illustratedin Table 2 by computing indexA and indexB from Eqs. (2) to (4) usingquantization parameters qPp and qPq for the neighboring blocks P and Q,respectively.

qPav=(qPp+qPq+1)>>1  (2)

indexA=Clip3(0, 51, qPav+FilterOffsetA)  (3)

indexB=Clip3(0, 51, qPav+FilterOffsetB)  (4)

TABLE 2 indexA (for α) or indexB (for β) 0 1 2 3 4 5 6 7 8 9 10 11 12 1314 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 α 0 0 0 0 0 0 0 0 0 0 00 0 0 0 0 4 4 5 6 7 8 9 10 12 13 15 17 20 22 β 0 0 0 0 0 0 0 0 0 0 0 0 00 0 0 2 2 2 3 3 3 3 4 4 4 6 6 7 7 indexA (for α) or indexB (for β) 30 3132 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 α 25 28 3236 40 45 50 56 63 71 80 90 101 113 127 144 162 182 203 226 255 255 β 8 89 9 10 10 11 11 12 12 13 13 14 14 15 15 16 16 17 17 18 18

Different filtering methods are defined for the case where “Bs<4” andthe case where “Bs=4”. First, the case where “Bs<4” will be described.

The deblocking filter conducts the operations indicated by Eqs. (5) to(7) and computes filtered pixel data p0′ and q0′. In Eq. (7), Clip3represents a clipping process.

p0′=Clip1(p0+Δ)  (5)

q0′=Clip1(q0+Δ)  (6)

Δ=Clip3(−tc, tc((((q0−p0)<<2)+(p1−q1)+4)>>3))  (7)

The deblocking filter computes “tc” in Eq. (7) on the basis of Eq. (8)in the case where chromaEdgeFlag is “0”, and on the basis of Eq. (9) inall other cases.

In Eq. (8), “( )?1:0” represents 1 if the condition in parenthesis ( )is satisfied, and represents “0” all other cases.

tc=tc0+((ap<β)?1:0)+(aq<β)?1:0)  (8)

tc=tc0+1  (9)

The value of tc is defined according to the values of Bs and indexA asin Table 3.

TABLE 3 indexA 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 2122 23 24 25 bS = 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1bS = 2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 bS = 3 0 0 00 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 indexA 26 27 28 29 30 3132 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 bS = 1 1 1 11 1 1 1 2 2 2 2 3 3 3 4 4 4 5 6 6 7 8 9 10 11 13 bS = 2 1 1 1 1 1 2 2 22 3 3 3 4 4 5 5 6 7 8 8 10 11 12 13 15 17 bS = 3 1 2 2 2 2 3 3 3 4 4 4 56 6 7 8 9 10 11 13 14 16 18 20 23 25

Also, the deblocking filter computes ap and aq in Eq. (8) in accordancewith Eqs. (10) and (11).

ap=|p2−p0|  (10)

aq=|q2−q0|  (11)

The deblocking filter computes filtered pixel data p1′ by conducting theoperations indicated in Eq. (12) in the case where chromaEdgeFlag is“0”, and in addition, ap is less than or equal to “β”. In all othercases, the filtered pixel data p1′ is obtained by Eq. (13).

p1′=p1+Clip3(−tc0, tc0, (p2+((p0+q0+1)>>1)−(p1<<1))>>1)  (12)

p1′=p1  (13)

The deblocking filter computes filtered pixel data q1′ by conducting theoperations indicated in Eq. (14) in the case where chromaEdgeFlag is“0”, and in addition, aq is less than or equal to “β”. In all othercases, the filtered pixel data q1′ is obtained by Eq. (15).

q1′=q1+Clip3(−tc0, tc0, (q2+((p0+q0+1)>>1)−(q1<<1))>>1)  (14)

q1′=q1  (15)

Next, the case where “Bs=4” will be described. The deblocking filtercomputes pixel data p0′, p1′, and p2′ in accordance with Eqs. (17) to(19) in the case where chromaEdgeFlag indicates “0”, and in addition,the condition in Eq. (16) is satisfied.

ap<β && |p0−q0|<((α>>2)+2)  (16)

p0′=(p2+2·p1+2·p0+2q0+q1+4)>>3  (17)

p1′=(p2+p1+p0+q0+2)>>2  (18)

p2′=(2·p3+3·p2+p1+p0+q0+4)>>3  (19)

The deblocking filter computes pixel data p0′, p1′, and p2′ inaccordance with Eqs. (20) to (22) in the case where chromaEdgeFlagindicates “0”, and in addition, the condition in Eq. (16) is notsatisfied.

p0′=(2·p1+p0+q1+2)>>2  (20)

p1′=p1  (21)

p2′=p2  (22)

The deblocking filter computes pixel data q0′, q1′, and q2′ inaccordance with Eqs. (24) to (26) in the case where chromaEdgeFlagindicates “0”, and in addition, the condition in Eq. (23) is satisfied.

aq<β && |p0−q0|<((α>>2)+2)  (23)

q0′=(p1+2·p0+2·q0+2·q1+q2+4)>>3  (24)

q1′=(p0+q0+q1+q2+2)>>2  (25)

q2′=(2·q3+3·q2+q1+q0+p4+4)>>3  (26)

The deblocking filter computes pixel data q0′, q1′, and q2′ inaccordance with Eqs. (27) to (29) in the case where chromaEdgeFlagindicates “0”, and in addition, the condition in Eq. (23) is notsatisfied.

q0′=(2·q1+q0+p1+2)>>2  (27)

q1′=q1  (28)

q2′=q2  (29)

<3. Configuration of Deblocking Filter and Filter Strength Adjuster inImage Encoding Apparatus>

The filter strength adjuster 41 adjusts the filter strength of thedeblocking filter 24 according to the prediction block size of theoptimal mode for a given macroblock.

Typically, blocking artifacts are more humanly visible in the case oflarger block sizes. Also, larger block sizes are more readily selectedfor flat areas that do not contain much texture information.

Also, although blocking artifacts are removed if the filter strength ofthe deblocking filter is strengthened, at the same time there is aproblem in that the sharpness of the image is lost since the filteringby a deblocking filter is a low-pass filter.

Accordingly, if for example it is configured such that the filter ismore readily applied to blocks using larger prediction block sizes whichdo not contain much texture information originally, then it becomespossible to remove easily visible blocking artifacts with little effecton texture, and obtain a decoded image with favorable image quality.

The filter strength adjuster 41 adjusts the filter strength of thedeblocking filter 24 by configuring parameter values that adjust thefilter strength according to the prediction block size.

Furthermore, it is not always the case that two neighboring blocks willhave equal prediction block sizes. Consequently, in the case where theprediction block size is an extended block size in the optimal mode foreither the target block or a neighboring block, the filter strengthadjuster 41 adjusts the filter strength according to the largerprediction block size.

Additionally, in the case where a plurality of macroblocks with a largerblock size than a given macroblock are used, the filter strengthadjuster 41 adjusts the filter strength according to a combination ofthe prediction block sizes for two neighboring blocks, such that thefilter strength is increased as the block size increases.

FIG. 4 illustrates a configuration of a deblocking filter and a filterstrength adjuster. The filter strength adjuster 41 is provided with ablock size buffer 411 and a parameter selector 412. The block sizebuffer 411 buffers one frame image's worth of information indicating theprediction block size of the optimal mode selected by the predictiveimage/optimal mode selector 33. In other words, a state is entered inwhich the block size buffer 411 stores information regarding theprediction block sizes of respective macroblocks in a single frame imageto be encoded.

The parameter selector 412 configures the parameter values on the basisof prediction block size information in the block size buffer 411, suchthat filtering is more readily applied to the parts of the decoded imagewith larger block sizes. The parameter selector 412 may for example setlarger values for the values of FilterOffsetA and FilterOffsetB, whichare specified by the syntax elements slice_alpha_c0_offset_div2 andslice_beta_offset_div2 in the H.264/AVC standard. The parameter selector412 also configures FilterOffsetA and FilterOffsetB in advance accordingto combinations of neighboring prediction block sizes, and determineswhat kind of prediction block sizes neighboring blocks have on the basisof prediction block size information. In addition, it is configured suchthat values which depend on combinations of determined prediction blocksizes are used.

Furthermore, the parameter selector 412 computes indexA and indexB fromthe earlier Eqs. (2) to (4) and uses the table illustrated in Table 2 toconfigure the threshold values α and β, i.e., the parameter values thatset the filter strength, according to the prediction block size, whichare then output to the filter strength determining unit 241 of thedeblocking filter 24.

The filter strength determining unit 241 determines block boundarystrength data Bs on the basis of prediction mode information suppliedfrom the lossless encoder 16, and outputs the determined block boundarystrength data Bs as well as the threshold values α and β supplied fromthe parameter selector 412 to the filter processor 242.

The filter processor 242 conducts the above-described filter operationsusing the block boundary strength data Bs, the threshold values α and β,and pixel data p3, p2, p1, p0, q0, q1, q2, and q3 at the block boundary,and computes filtered pixel data p3′, p2′, p1′, p0′, q0′, q1′, q2′, andq3′.

<4. Operation of Image Encoding Apparatus>

Next, image encoding processing operations will be described. FIG. 5illustrates prediction block sizes used in an image encoding process. Inthe H.264/AVC format, prediction block sizes from 16×16 pixels to 4×4pixels are defined as illustrated by (C) and (D) in FIG. 5. Also, in thecase of using macroblocks with sizes extended beyond that of theH.264/AVC format, such as the case of using 32×32 pixel macroblocks, forexample, the prediction block sizes illustrated by (B) in FIG. 5 aredefined, for example. Also, in the case of using 64×64 pixelmacroblocks, for example, the prediction block sizes illustrated by (A)in FIG. 5 are defined, for example.

In FIG. 5, “skip/direct” indicates a prediction block size when skippedmacroblocks or direct mode is selected in the motionprediction/compensation unit 32. Also, “ME” indicates a motioncompensation block size. Also, “P8×8” indicates that a macroblock can bedivided further at a lower layer with macroblocks of smaller size.

Meanwhile, standardization of a coding format called HEVC (HighEfficiency Video Coding) has been progressing under work by the JCTVC(Joint Collaboration Team-Video Coding), a joint standardization groupbetween the ITU-T and the ISO/IEC, with the aim of further improving thecoding efficiency over the H.264/AVC format. As of September 2010, “TestModel under Consideration” (JCTVC-B205) has been published as a draft.

With the HEVC coding format, a CU (Coding Unit) is defined, making itpossible to use macroblocks with sizes extended beyond that ofH.264/AVC.

A CU, also called a CTB (Coding Tree Block), fulfills a similar role toa macroblock in the H.264/AVC format. Also, whereas macroblocks arefixed at a size of 16×16 pixels in the H.264/AVC format, with the HEVCcoding format the size is not fixed and is specified in the imagecompression information in respective sequences.

Particularly, the CU with the largest size is called the LCU (LargestCoding Unit). Also, the CU with the smallest size is called the SCU(Smallest Coding Unit). While these sizes are specified in the sequenceparameter set contained in the image compression information, they areeach limited to being square sizes expressed by a power of 2.

FIG. 6 illustrates exemplary CUs defined by the HEVC coding format. Inthe example illustrated, the size of the LCU is 128, and the maximumlayer depth is 5. When the value of split_flag is 1, a CU of size 2N×2Nis split into CUs of size N×N one layer below.

Additionally, a CU is split into PUs (Prediction Units), which are theunits of intra or inter prediction, and TUs (Transform Units), which arethe units of orthogonal transform.

A CU is additionally split into PUs, which are the units of intra orinter prediction, or alternatively TUs, which are the units oforthogonal transform, and a prediction process and orthogonal transformprocess are conducted. At present, with the HEVC coding format it ispossible to use 16×16 as well as 32×32 orthogonal transform in additionto 4×4 and 8×8.

In this specification, blocks and macroblocks include the concepts ofCUs, PUs, and TUs as discussed above, and are not limited to blocks offixed size.

FIG. 7 is a flowchart illustrating image encoding processing operations.In step ST11, the A/D converter 11 A/D converts an input image signal.

In step ST12, the frame sort buffer 12 sorts images. The frame sortbuffer 12 stores image data supplied by the A/D converter 11, resortingpictures from their display order to an encoding order.

In step ST13, the subtractor 13 generates prediction error data. Thesubtractor 13 generates prediction error data by computing thedifference between the image data of an image sorted in step ST12 andpredictive image data supplied by the predictive picture/optimal modeselector 33. Prediction error data has a smaller data size compared tothe original image data. Consequently, the data size can be compressedcompared to the case of encoding images directly. Herein, when theselection between a predictive image supplied from the intra predictor31 and a predictive image supplied from the motionprediction/compensation unit 32 is conducted in units of slices by thepredictive image/optimal mode selector 33, intra prediction is conductedon slices for which a predictive image supplied from the intra predictor31 is selected. Meanwhile, inter prediction is conducted on slices forwhich a predictive image from the motion prediction/compensation unit 32is selected.

In step ST14, the orthogonal transform unit 14 applies an orthogonaltransform. The orthogonal transform unit 14 applies an orthogonaltransform to prediction error data supplied from the subtractor 13.Specifically, an orthogonal transform such as the discrete cosinetransform or the Karhunen-Loeve transform is applied to prediction errordata, and transform coefficient data is output.

In step ST15, the quantizer 15 conducts quantization. The quantizer 15quantizes transform coefficient data. During quantization, rate controlis applied as described later in the processing in step ST25.

In step ST16, the inverse quantizer 21 conducts inverse quantization.The inverse quantizer 21 inverse quantizes transform coefficient datathat has been quantized by the quantizer 15, using characteristics thatcorrespond to the characteristics of the quantizer 15.

In step ST17, the inverse orthogonal transform unit 22 applies aninverse orthogonal transform. The inverse orthogonal transform unit 22applies an inverse orthogonal transform to transform coefficient datathat has been inverse quantized by the inverse quantizer 21, usingcharacteristics that correspond to the characteristics of the orthogonaltransform unit 14.

In step ST18, the adder 23 generates decoded image data. The adder 23generates decoded image data by adding together predictive image datasupplied from the predictive image/optimal mode selector 33 and inverseorthogonally transformed data at a position corresponding to thatpredictive image.

In step ST19, the deblocking filter 24 applies filtering. The deblockingfilter 24 filters decoded image data output by the adder 23 to removeblocking artifacts.

In step ST20, the frame memory 25 stores decoded image data. The framememory 25 stores unfiltered decoded image data and filtered decodedimage data.

In step ST21, the intra predictor 31 and the motionprediction/compensation unit 32 conduct respective prediction processes.Namely, the intra predictor 31 conducts intra prediction in intraprediction modes, while the motion prediction/compensation unit 32conducts motion prediction/compensation in inter prediction modes.Details of the prediction processes will be discussed later withreference to FIG. 8, but as a result of these processes, prediction isconducted in all candidate prediction modes, and cost function valuesare respectively computed for all candidate prediction modes. Then, anoptimal intra prediction mode and an optimal inter prediction mode areselected on the basis of the computed cost function values, andpredictive images generated with the selected prediction modes as wellas their cost function values and prediction mode information aresupplied to the predictive image/optimal mode selector 33.

In step ST22, the predictive image/optimal mode selector 33 selectspredictive image data. The predictive image/optimal mode selector 33determines the optimal mode with the best encoding efficiency on thebasis of the respective cost function values output by the intrapredictor 31 and the motion prediction/compensation unit 32.Additionally, the predictive image/optimal mode selector 33 selects thepredictive image of the optimal mode thus determined, and supplies it tothe subtractor 13 and the adder 23. This predictive image is used in theoperations in steps ST13 and ST18, as discussed above. Meanwhile, theprediction mode information corresponding to the selected predictiveimage data is output to the lossless encoder 16 and the filter strengthadjuster 41.

In step ST23, the lossless encoder 16 conducts lossless encoding. Thelossless encoder 16 losslessly encodes quantized data output by thequantizer 15. In other words, data is compressed by applying losslessencoding such as variable-length coding or arithmetic coding toquantized data. At this point, the prediction mode information inputinto the lossless encoder 16 in step ST22 discussed above (which mayinclude the macroblock type, prediction mode, motion vector information,and reference picture information, for example) is also losslesslyencoded. Furthermore, the losslessly encoded prediction mode informationdata is added to the header information of the encoded stream generatedby losslessly encoding the quantized data.

In step ST24, the accumulation buffer 17 conducts a buffering process tobuffer an encoded stream. The encoded stream buffered in theaccumulation buffer 17 is read out as appropriate and transmitted to thedecoder via a transmission channel.

In step ST25, the rate controller 18 applies rate control. The ratecontroller 18 controls the rate of quantization operations by thequantizer 15 such that overflows or underflows do not occur in theaccumulation buffer 17 when an encoded stream is being buffered in theaccumulation buffer 17.

Next, the prediction process in step ST21 of FIG. 7 will be describedwith reference to the flowchart in FIG. 8.

In step ST31, the intra predictor 31 conducts intra prediction. Theintra predictor 31 intra predicts processing target block images in allcandidate intra prediction modes. Herein, decoded image data stored inthe frame memory 25 that has not been filtered by the deblocking filter24 is used as the image data of a decoded image referenced during intraprediction. Details of the intra prediction process will be discussedlater, but as a result of this process, intra prediction is conducted inall candidate intra prediction modes, and cost function values arecomputed for all candidate intra prediction modes. Then, the one intraprediction mode with the best encoding efficiency is selected from amongall intra prediction modes on the basis of the computed cost functionvalues.

In step ST32, the motion prediction/compensation unit 32 conducts interprediction. The motion prediction/compensation unit 32 uses filtereddecoded image data stored in the frame memory 25 to inter predict in allcandidate inter prediction modes (all prediction block sizes). Detailsof the inter prediction process will be discussed later, but as a resultof this process, prediction processing is conducted in all candidateinter prediction modes, and cost function values are computed for allcandidate inter prediction modes. Then, the one inter prediction modewith the best encoding efficiency is selected from among all interprediction modes on the basis of the computed cost function values.

Next, the intra prediction process in step ST31 of FIG. 8 will bedescribed with reference to the flowchart in FIG. 9.

In step ST41, the intra predictor 31 conducts intra prediction inrespective prediction modes. The intra predictor 31 uses unfiltereddecoded image data stored in the frame memory 25 to generate predictiveimage data for each intra prediction mode.

In step ST42, the intra predictor 31 computes a cost function value foreach prediction mode. Computation of a cost function value is conductedon the basis of a technique in either a high-complexity mode or alow-complexity mode, as defined by the JM (Joint Model), the referencesoftware for the H.264/AVC format.

In other words, in the high-complexity mode, the processing in step ST41involves provisionally conducting lossless encoding in all candidateprediction modes, and computing a cost function value expressed by thefollowing Eq. (30) for each prediction mode.

Cost (ModeεΩ)=D+λ·R  (30)

Ω represents the set of all candidate prediction modes for encoding theblocks and macroblocks. D represents the differential energy(distortion) between a decoded image that has been encoded in aprediction mode, and an input image. R is the bit rate, including theorthogonal transform coefficients and prediction mode information, etc.λ is the Lagrange multiplier given as a function of the quantizationparameter QP.

In other words, since encoding in the high-complexity mode involvescomputing the above parameters D and R, it is necessary to provisionallyencode once in all candidate prediction modes, which is morecomputationally intensive.

Meanwhile, in the low-complexity mode, the processing in step ST41involves generating a predictive image and computing header bits such asmotion vector information and prediction mode information for allcandidate prediction modes. Additionally, a cost function valueexpressed by the following Eq. (31) is computed for each predictionmode.

Cost(ModeεΩ)=D+QPtoQuant (QP)·Header_Bit  (31)

Ω represents the set of all candidate prediction modes for encoding theblocks and macroblocks. D represents the differential energy(distortion) between a decoded image that has been encoded in aprediction mode, and an input image. Header_Bit is the number of headerbits for the prediction mode, and QPtoQuant is a function given as afunction of the quantization parameter QP.

In other words, although in the low-complexity mode it is necessary toconduct a prediction process for each prediction mode, a decoded imageis not necessary, and thus it is possible to realize the low-complexitymode with less computation than the high-complexity mode.

In step ST43, the intra predictor 31 determines the optimal intraprediction mode. The intra predictor 31 determines the optimal intraprediction mode by selecting the one intra prediction mode whose costfunction value yields the minimum value from among the cost functionvalues computed in step ST42.

Next, the inter prediction process in step ST32 of FIG. 8 will bedescribed with reference to the flowchart in FIG. 10.

In step ST51, the motion prediction/compensation unit 32 respectivelydetermines a motion vector and a reference image for each predictionmode. In other words, the motion prediction/compensation unit 32respectively determines a motion vector and a reference image for atarget block in each prediction mode.

In step ST52, the motion prediction/compensation unit 32 applies motioncompensation in each prediction mode. For each prediction mode (eachprediction block size), the motion prediction/compensation unit 32applies motion compensation to a reference image on the basis of amotion vector determined in step ST51, and generates predictive imagedata for each prediction mode.

In step ST53, the motion prediction/compensation unit 32 generatesmotion vector information in each prediction mode. The motionprediction/compensation unit 32 generates motion vector information toinclude in the encoded stream for a motion vector determined for eachprediction mode. For example, a predictive motion vector may bedetermined using median prediction, etc., and motion vector informationexpressing the difference between a motion vector detected by motionprediction and the predictive motion vector may be generated. Motionvector information generated in this way is also used to compute a costfunction value in the following step ST54, and in the case where itscorresponding predictive image is ultimately selected by the predictiveimage/optimal mode selector 33, the motion vector information isincluded in the prediction mode information and output to the losslessencoder 16.

In step ST54, the motion prediction/compensation unit 32 computes a costfunction value for each inter prediction mode. The motionprediction/compensation unit 32 computes a cost function value using Eq.(30) or (31) discussed earlier. Herein, the computation of a costfunction value for an inter prediction mode also includes the costfunction value scoring for skip mode and direct mode defined in theH.264/AVC format.

In step ST55, the motion prediction/compensation unit 32 determines theoptimal inter prediction mode. The motion prediction/compensation unit32 determines the optimal inter prediction mode by selecting the oneprediction mode whose cost function value yields the minimum value fromamong the cost function values computed in step ST54.

Next, a deblocking filter adjustment process will be described. In themethod of stipulating parameters regarding a deblocking filter in theH.264/AVC coding format, there exists adeblocking_filter_control_present_flag in the picture parameter set. Itis configured such that slice_alpha_c0_offset_div2 and slice_offset_div2exist in the slice header in the case where this flag has a value of“1”, whereas a default value of “0” is used in the case where they donot exist.

In contrast, in the present invention, the values of FilterOffsetA andFilterOffsetB are both defined according to the combination of twoneighboring block sizes as in Table 4, in which offset values SZa andSZb are taken to be integers such that 0<SZa≦SZb≦6, for example.

TABLE 4 64 × 64 32 × 32 16 × 16 64 × 64 SZb SZb SZb 32 × 32 SZb SZa SZa16 × 16 SZb SZa 0

Table 4 illustrates offset values for individual combinations of theprediction block sizes for two neighboring blocks such that the filterstrength is increased as the block size increases, in the case where aplurality of macroblocks with a larger block size than a givenmacroblock are used. Also, the offset values in Table 4 are set suchthat the filter strength is adjusted according to the larger predictionblock size in the case where the two neighboring blocks have differentprediction block sizes. In other words, the values of FilterOffsetA andFilterOffsetB are configured to match the block of larger size frombetween the two neighboring blocks. For example, the offset value SZb isused when the larger block is 64 pixels, the offset value SZa is usedwhen the larger block is 32 pixels, and an offset value of “0” is usedfor 16×16 pixels.

FIG. 11 is a flowchart illustrating a filter configuration process. Instep ST61, the filter strength adjuster 41 acquires the prediction blocksize of the optimal mode. The filter strength adjuster 41 acquires theprediction block size corresponding to the predictive image selected instep ST22 of FIG. 7, or in other words, the prediction block size usedwhen encoding in the optimal mode.

In step ST62, the filter strength adjuster 41 determines whether or notthe boundary portion of the current block or neighboring block is 64pixels. The filter strength adjuster 41 proceeds to step ST63 when theboundary potion of the current block of neighboring block is 64 pixels,and proceeds to step ST64 when the boundary portion of the current blockand the neighboring block are both smaller than 64 pixels.

In step ST63, the filter strength adjuster 41 selects the offset valueSZb, sets it as FilterOffsetA and FilterOffsetB, and proceeds to stepST67.

In step ST64, the filter strength adjuster 41 determines whether or notthe boundary portion of the current block or neighboring block is 32pixels. The filter strength adjuster 41 proceeds to step ST65 when theboundary potion of the current block of neighboring block is 32 pixels,and proceeds to step ST66 when the boundary portion of the current blockand the neighboring block are both smaller than 32 pixels.

In step ST65, the filter strength adjuster 41 selects the offset valueSZa (SZb), sets it as FilterOffsetA and FilterOffsetB, and proceeds tostep ST67.

In step ST66, the filter strength adjuster 41 selects the offset value 0(<SZa), sets it as FilterOffsetA and FilterOffsetB, and proceeds to stepST67.

In step ST67, the filter strength adjuster 41 adjusts the filterstrength. The filter strength adjuster 41 uses FilterOffsetA andFilterOffsetB configured in step ST63, 65, or 66 to determine thethreshold values α and β, i.e., the parameter values that set the filterstrength, and outputs them to the deblocking filter 24.

In this way, as a result of the filter strength adjuster 41 configuringthe threshold values α and β with increased offset values when the blocksize at the boundary portion is large, filtering is more readily appliedby the deblocking filter 24.

Note that although combinations of 64×64 pixels, 32×32 pixels, and 16×16pixels are illustrated by example in Table 4 and FIG. 11, combinationsare not limited to these sizes. Also, although the offset value SZb isused when the larger block is 64 pixels, the offset value SZa is usedwhen the larger block is 32 pixels, and an offset value of “0” is usedfor 16×16 pixels, offset values are not limited to the abovecombinations, as long as favorable image quality is obtained by havingthe filter be more readily applied to larger blocks.

Furthermore, in the case where the user wants to configure his or herown value different from Table 4, it is possible for the user toconfigure the values of FilterOffsetA16, FilterOffsetA32,FilterOffsetA64, FilterOffsetB16, FilterOffsetB32, and FilterOffsetB64in the slice header, with a given value of “1” set as the value ofdeblocking_filter_control_present_flag in the picture parameter set.These respective parameters correspond to the cases where the larger ofthe two neighboring blocks is 16×16, 32×32, and 64×64. Values forFilterOffsetA32, FilterOffsetA64, FilterOffsetB32, and FilterOffsetB64need not exist in the slice header, even if the value ofdeblocking_filter_control_present_flag is 1. In this case, values suchas

FilterOffsetA32=FilterOffsetA16+SZa

FilterOffsetA64=FilterOffsetA16+SZb

FilterOffsetB32=FilterOffsetB16+SZa

FilterOffsetB64=FilterOffsetB16+SZb

may be used.

In this way, according to an image encoding apparatus and method towhich the present invention has been applied, the block size yieldingthe best encoding efficiency is determined, and image data is encodedusing the determined block size. At this point, information expressingthe block size is buffered in the block size buffer 411 of the filterstrength adjuster 41. Consequently, when image data that has beenencoded using the block size yielding the best encoding efficiency isdecoded and decoded image data is generated, the positions of predictionblocks in the decoded image are clear. For this reason, if the filterstrength is adjusted on the basis of information buffered in the blocksize buffer 411 such that the filter is applied more readily to largeblock sizes, blocking artifacts can be reduced even if the block size islarge. Also, since blocking artifacts can be reduced in a decoded image,the encoded data size can be further reduced, since it is possible toprevent increases in prediction error data due to the effects ofblocking artifacts.

<5. Configuration of Image Decoding Apparatus>

An encoded stream that has been generated by encoding an input image issupplied to an image decoding apparatus via a given transmission channelor recording medium, etc. and decoded.

FIG. 12 illustrates a configuration of an image decoding apparatus. Theimage decoding apparatus 50 is provided with an accumulation buffer 51,a lossless decoder 52, an inverse quantizer 53, an inverse orthogonaltransform unit 54, an adder 55, a deblocking filter 56, a frame sortbuffer 57, and a D/A converter 58. Additionally, the image decodingapparatus 50 is provided with frame memory 61, selectors 62 and 65, anintra predictor 63, a motion compensation unit 64, and a filter strengthadjuster 71.

The accumulation buffer 51 buffers an encoded stream transmittedthereto. The lossless decoder 52 decodes an encoded stream supplied bythe accumulation buffer 51 in a format corresponding to the encodingformat of the lossless encoder 16 in FIG. 1. Also, the lossless decoder52 decodes header information in the encoded stream to obtain predictionmode information, which is output to the intra predictor 63, the motioncompensation unit 64, and the deblocking filter 56.

The inverse quantizer 53 takes quantized data that has been decoded bythe lossless decoder 52 and inverse quantizes it in a formatcorresponding to the quantization format of the quantizer 15 in FIG. 1.The inverse orthogonal transform unit 54 takes the output from theinverse quantizer 53, applies an inverse orthogonal transform in aformat corresponding to the orthogonal transform format of theorthogonal transform unit 14 in FIG. 1, and outputs the result to theadder 55.

The adder 55 adds together inverse orthogonally transformed data andpredictive image data supplied from the selector 65 to generate decodedimage data, which is output to the deblocking filter 56 and the framememory 61.

The deblocking filter 56 filters decoded image data supplied from theadder 55 to remove blocking artifacts, and then supplies the result tothe frame memory 61 for buffering, while also outputting the result tothe frame sort buffer 57. The deblocking filter 56 also adjusts thefilter strength on the basis of prediction mode information suppliedfrom the lossless decoder 52 and threshold values α and β for filterstrength adjustment supplied from the filter strength adjuster 71discussed later.

The frame sort buffer 57 sorts images. In other words, the framesequence that was resorted into an encoding order by the frame sortbuffer 12 in FIG. 1 is resorted into the original display order andoutput to the D/A converter 58.

The D/A converter 58 D/A converts image data supplied from the framesort buffer 57 and causes an image to be displayed by outputting theimage data to a display not illustrated in the drawings.

The frame memory 61 stores the unfiltered decoded image data suppliedfrom the adder 55 and the filtered and decoded image data supplied fromthe deblocking filter 24.

The selector 62, on the basis of prediction mode information suppliedfrom the lossless decoder 52, supplies unfiltered decoded image dataread out from the frame memory 61 to the intra predictor 63 whendecoding a prediction block that has been intra predicted. Also, theselector 26, on the basis of prediction mode information supplied fromthe lossless decoder 52, supplies filtered and decoded image data readout from the frame memory 61 to the motion compensation unit 64 whendecoding a prediction block that has been inter predicted.

The intra predictor 63 generates a predictive image on the basis ofprediction mode information supplied from the lossless decoder 52, andoutputs the generated predictive image data to the selector 65. Theintra predictor 63 also outputs information indicating the block sizesof the generated predictive image to the filter strength adjuster 71.

The motion compensation unit 64 applies motion compensation andgenerates predictive image data on the basis of prediction modeinformation supplied from the lossless decoder 52, and outputs theresult to the selector 65. In other words, on the basis of motion vectorinformation and reference frame information contained in the predictionmode information, the motion compensation unit 64 generates predictiveimage data by applying motion compensation to the reference imageindicated by the reference frame information using motion vectors basedon the motion vector information. The motion compensation unit 64 alsooutputs information indicating the block sizes of the generatedpredictive image to the filter strength adjuster 71.

The selector 65 supplies predictive image data generated by the intrapredictor 63 to the adder 55. The selector 65 also supplies predictiveimage data generated by the motion compensation unit 64 to the adder 55.

The filter strength adjuster 71 is configured similarly and operatessimilarly to the filter strength adjuster 41 illustrated in FIG. 4. Thefilter strength adjuster 71 adjusts the filter strength of thedeblocking filter 56 by configuring parameter values that adjust thefilter strength according to the prediction block size. Also, in thecase where the prediction block size is an extended block size in theoptimal mode for either the target block or a neighboring block, thefilter strength adjuster 71 adjusts the filter strength according to thelarger prediction block size. Additionally, in the case where aplurality of macroblocks with a larger block size than a givenmacroblock are used, the filter strength adjuster 71 adjusts the filterstrength according to a combination of the prediction block sizes fortwo neighboring blocks, such that the filter strength is increased asthe block size increases. In other words, the filter strength adjuster71 configures the threshold values α and β, i.e., the parameter valuesfor adjusting the filter strength, according to the block size suppliedfrom the intra predictor 63 or the motion compensation unit 64, andoutputs the configured parameter values to the deblocking filter 56.

<6. Operation of Image Decoding Apparatus>

Next, image decoding processing operations conducted by the imagedecoding apparatus 50 will be described with reference to the flowchartin FIG. 13.

In step ST71, the accumulation buffer 51 buffers an encoded streamtransmitted thereto. In step ST72, the lossless decoder 52 conductslossless decoding. The lossless decoder 52 decodes an encoded streamsupplied from the accumulation buffer 51. In other words, quantized datafor respective pictures that have been encoded by the lossless encoder16 in FIG. 1 are obtained. Also, the lossless decoder 52 losslesslydecodes prediction mode information contained in the header informationof the encoded stream, and supplies the obtained prediction modeinformation to the deblocking filter 56 as well as the selectors 62 and65. Furthermore, in the case where the prediction mode information isinformation regarding an intra prediction mode, the lossless decoder 52outputs the prediction mode information to the intra predictor 63. Also,in the case where the prediction mode information is informationregarding an inter prediction mode, the lossless decoder 52 outputs theprediction mode information to the motion compensation unit 64.

In step ST73, the inverse quantizer 53 conducts inverse quantization.The inverse quantizer 53 takes quantized data that has been decoded bythe lossless decoder 52 and inverse quantizes it using characteristicsthat correspond to the characteristics of the quantizer 15 in FIG. 1.

In step ST74, the inverse orthogonal transform unit 54 applies aninverse orthogonal transform. The inverse orthogonal transform unit 54applies an inverse orthogonal transform to transform coefficient datathat has been inverse quantized by the inverse quantizer 53, usingcharacteristics that correspond to the characteristics of the orthogonaltransform unit 14 in FIG. 1.

In step ST75, the adder 55 generates decoded image data. The adder 55generates decoded image data by adding together data obtained byapplying the inverse orthogonal transform, and predictive image dataselected in step ST79 to be discussed later. In so doing, an originalimage is decoded.

In step ST76, the deblocking filter 56 applies filtering. The deblockingfilter 56 filters decoded image data output by the adder 55 to removeblocking artifacts contained in the decoded image.

In step ST77, the frame memory 61 conducts a decode image data storageprocess.

In step ST78, the intra predictor 63 and the motion compensation unit 64conduct prediction processes. The intra predictor 63 and the motioncompensation unit 64 conduct respective prediction processescorresponding to prediction mode information supplied from the losslessdecoder 52.

In other words, in the case where prediction mode information for intraprediction is supplied from the lossless decoder 52, the intra predictor63 conducts an intra prediction process on the basis of the predictionmode information, and generates predictive image data. Also, in the casewhere prediction mode information for inter prediction is supplied fromthe lossless decoder 52, the motion compensation unit 64 applies motioncompensation on the basis of the prediction mode information, andgenerates predictive image data.

In step ST79, the selector 65 selects predictive image data. In otherwords, the selector 65 selects between a predictive image supplied fromthe intra predictor 63 and predictive image data generated by the motioncompensation unit 64, and supplies the selection to the adder 55, whichcauses it to be added to the output from the inverse orthogonaltransform unit 54 in step ST75, as discussed above.

In step ST80, the frame sort buffer 57 sorts images. In other words, theframe sort buffer 57 takes the frame sequence that was resorted into anencoding order by the frame sort buffer 12 of the image encodingapparatus 10 in FIG. 1, and resorts it into the original display order.

In step ST81, the D/A converter 58 D/A converts image data from theframe sort buffer 57. The image is output to a display not illustratedin the drawings, and the image is displayed.

Also, with the filter process in step ST76 of FIG. 13, the filterstrength adjuster 71 conducts the filter strength adjustment processdiscussed earlier and illustrated in FIG. 11. The filter strengthadjuster 71 configures the threshold values α and β, i.e., the parametervalues that set the filter strength, such that the filter is morereadily applied to larger prediction block sizes by adjusting the valuesof FilterOffsetA and FilterOffsetB, which are specified by the syntaxelements slice_alpha_c0_offset_div2 and slice_beta_offset_div2 in theH.264/AVC standard, for example.

In this way, the filter strength adjuster 71 generates the thresholdvalues α and β, i.e., the parameter values for adjusting the filterstrength, according to the prediction block size from when encoding wasconducted, and outputs them to the deblocking filter 56. The deblockingfilter 56 determines block boundary strength data Bs on the basis ofprediction mode information and uses the threshold values α and β tofilter decoded image data.

Also, when the value of the syntax elementdeblocking_filter_control_present_flag in the picture parameter set isset to a given value and values for FilterOffsetA and FilterOffsetB arestated in the slice header, the filter strength adjuster 71 uses thestated values to generate the threshold values α and β and output themto the deblocking filter 56. If such filter strength adjustment isconducted, it becomes possible to apply filtering according to usersettings even in the image decoding apparatus in the case where a userconfigured his or her own values different from those in Table 4 in theimage encoding apparatus.

In this way, according to an image decoding apparatus and method towhich the present invention has been applied, information indicating theblock size used in the encoding process is buffered in a block sizebuffer of the filter strength adjuster 71. Consequently, when decodedimage data is generated by decoded an encoded stream, the positions ofprediction blocks in the decoded image are clear. For this reason, ifthe filter strength is adjusted on the basis of information buffered inthe block size buffer such that the filter is applied more readily tolarge block sizes, blocking artifacts can be reduced even if the blocksize is large. Consequently, it becomes possible to obtain a decodedimage with favorable image quality.

In addition, it is possible to execute the processes described in thisspecification in hardware, in software, or in a compound configurationof both. In the case of executing processes in software, a programstating a processing sequence may be installed onto memory in a computerbuilt into special-purpose hardware and executed. Alternatively, it isalso possible to install and execute the program on a general-purposecomputer capable of executing various processes.

For example, the program may be recorded in advance onto a recordingmedium in the form of a hard disk or ROM (Read-Only Memory).Alternatively, the program may be temporarily or permanently stored(recorded) on a removable recording medium such as a flexible disk,CD-ROM (Compact Disc-Read-Only Memory), MO (Magneto-Optical) disc, DVD(Digital Versatile Disc), magnetic disk, or semiconductor memory. Such aremovable recording medium may be provided in the form of what is calledpackaged media.

Meanwhile, besides being installed onto a computer from a removablerecording medium as described above, the program may also be wirelesslytransferred to a computer from a download site or transferred to acomputer in a wired manner via a network such as a LAN (Local AreaNetwork) or the Internet. On the computer, the program transferred inthe above manner may be received and installed onto a recording mediumsuch as an internal hard disk.

Furthermore, the present invention is not to be interpreted as beinglimited to the foregoing embodiments of the present invention. Theembodiments of the invention disclose the present invention in anexemplary and illustrative form, and it should be obvious thatmodifications and substitutions may occur to persons skilled in the artwithout departing from the principal matter of the present invention. Inother words, the principal matter of the present invention should bedetermined in consideration of the claims.

INDUSTRIAL APPLICABILITY

With an image processing apparatus and image processing method of theinvention, the filter strength is adjusted according to the block sizesof neighboring blocks adjacent at a block boundary when applyingfiltering to remove blocking artifacts from decoded image data. For thisreason, the filter strength is adjusted according to the block size evenif macroblocks of extended size are used, and thus a decoded image withreduced blocking artifacts and favorable image quality is obtained.Consequently, the present invention is suitable for imaging encodingapparatus and image decoding apparatus used when image information (abit stream) obtained by encoding in units of blocks as with MPEG andH.26x is transmitted and received via network media such as satellitebroadcasting, cable TV, the Internet, and mobile phones, or processed onrecording media such as optical or magnetic disks and flash memory.

REFERENCE SIGNS LIST

-   -   10 image encoding apparatus    -   11 A/D converter    -   12, 57 frame sort buffer    -   13 subtractor    -   14 orthogonal transform unit    -   15 quantizer    -   16 lossless encoder    -   17, 51 accumulation buffer    -   18 rate controller    -   21, 53 inverse quantizer    -   22, 54 inverse orthogonal transform unit    -   23, 55 adder    -   24 deblocking filter    -   25, 61 frame memory    -   26, 62, 65 selector    -   31, 63 intra predictor    -   32 motion prediction/compensation unit    -   33 predictive image/optimal mode selector    -   41 filter strength adjuster    -   50 image decoding apparatus    -   52 lossless decoder    -   58 D/A converter    -   64 motion compensation unit    -   71 filter strength adjuster    -   241 filter strength determining unit    -   242 filter processor    -   411 block size buffer    -   412 parameter selector

1-20. (canceled)
 21. An image processing apparatus comprising: a decoderthat generates decoded image data by decoding an encoded stream in whichimage data has been encoded in individual blocks; a filter that appliesfiltering to remove blocking artifacts from neighboring blocks adjacentat a block boundary in the decoded image data generated by the decoder;and a filter strength adjuster that uses parameters for adjusting thefilter strength of the filtering to adjust the filter strength set forneighboring blocks whose block sizes are a greater size than a givenblock size.
 22. The image processing apparatus according to claim 21,wherein the filter strength adjuster adjusts the filter strength usingparameters configured as offset values for adjusting the filterstrength.
 23. The image processing apparatus according to claim 22,wherein the offset values are configured according to the offset valuesand the block sizes of the neighboring blocks.
 24. The image processingapparatus according to claim 23, wherein the offset values areconfigured according to the combination of block sizes of theneighboring blocks.
 25. The image processing apparatus according toclaim 24, wherein the offset values are configured according to thelarger block size in a case that the block sizes differ between twoneighboring blocks.
 26. The image processing apparatus according toclaim 23, wherein the offset values are configured to increase as theblock sizes of the neighboring blocks increase.
 27. The image processingapparatus according to claim 26, wherein the filter strength adjusterselects the offset values according to the block sizes of theneighboring blocks from offset values configured for each block size ofthe neighboring blocks.
 28. The image processing apparatus according toclaim 21, wherein the given size is a size of 16×16 pixels, being themacroblock size of AVC standard.
 29. An image processing methodcomprising: a step that generates decoded image data by decoding anencoded stream in which image data has been encoded in individualblocks; a step that applies filtering to remove blocking artifacts fromthe decoded image data generated by the decoder; and a step that adjuststhe filter strength of the filtering according to the block sizes ofneighboring blocks adjacent at a block boundary.
 30. An image processingapparatus comprising: a filter that applies filtering to remove blockingartifacts from neighboring blocks adjacent at a block boundary indecoded image data that has been locally decoded; a filter strengthadjuster that uses parameters for adjusting the filter strength of thefiltering to adjust the filter strength of the filtering set forneighboring blocks whose block sizes are a greater size than a givenblock size; and an encoder that encodes image data by using decodedimage data that has been filtered by the filter.
 31. The imageprocessing apparatus according to claim 30, wherein the filter strengthadjuster configures the parameters as offset values for adjusting thefilter strength.
 32. The image processing apparatus according to claim31, wherein the filter strength adjuster configures the offset valuesaccording to the block sizes of the neighboring blocks.
 33. The imageprocessing apparatus according to claim 32, wherein the filter strengthadjuster configures the offset values according to the combination ofblock sizes of the neighboring blocks.
 34. The image processingapparatus according to claim 33, wherein the filter strength adjusterconfigures the offset values according to the larger block size in thecase where the block sizes differ between two neighboring blocks. 35.The image processing apparatus according to claim 32, wherein the filterstrength adjuster configures the offset values to increase as the blocksizes of the neighboring blocks increase.
 36. The image processingapparatus according to claim 25, wherein the filter strength adjusterconfigures the offset values for each block size of the neighboringblocks, and selects the offset values according to the block sizes ofthe neighboring blocks.
 37. The image processing apparatus according toclaim 30, wherein the given size is a size of 16×16 pixels, being themacroblock size of AVC standard.
 38. An image processing methodcomprising: a step that applies filtering to remove blocking artifactsfrom decoded image data; a step that adjusts the filter strength of thefiltering according to the block sizes of neighboring blocks adjacent ata block boundary; and a step that encodes image data by using decodedimage data that has been filtered by the filter.